Method and system for considering information about an expected response when performing speech recognition

ABSTRACT

A speech recognition system receives and analyzes speech input from a user in order to recognize and accept a response from the user. Under certain conditions, information about the response expected from the user may be available. In these situations, the available information about the expected response is used to modify the behavior of the speech recognition system by taking this information into account. The modified behavior of the speech recognition system comprises adjusting the rejection threshold when speech input matches the predetermined expected response.

RELATED APPLICATION

This application is a continuation Application of U.S. Ser. No.11/051,825, now U.S. Patent Publication No. 2006-0178882, filed Feb. 4,2005, entitled “METHOD AND SYSTEM FOR CONSIDERING INFORMATION ABOUT ANEXPECTED RESPONSE WHEN PERFORMING SPEECH RECOGTNIION”.

TECHNICAL FIELD

The present invention relates to speech recognition and, moreparticularly, to speech recognition systems for addressing likely orexpected responses.

BACKGROUND ART

Speech recognition has simplified many tasks in the workplace bypermitting hands-free communication with a computer as a convenientalternative to communication via conventional peripheral input/outputdevices. A worker may enter data by voice using a speech recognizer andcommands or instructions may be communicated to the worker by a speechsynthesizer. Speech recognition finds particular application in mobilecomputing devices in which interaction with the computer by conventionalperipheral input/output devices is restricted.

For example, wireless wearable terminals can provide a worker performingwork-related tasks with desirable computing and data-processingfunctions while offering the worker enhanced mobility within theworkplace. One particular area in which workers rely heavily on suchwireless wearable terminals is inventory management. Inventory-drivenindustries rely on computerized inventory management systems forperforming various diverse tasks, such as food and retail productdistribution, manufacturing, and quality control. An overall integratedmanagement system involves a combination of a central computer systemfor tracking and management, and the people who use and interface withthe computer system in the form of order fillers, pickers and otherworkers. The workers handle the manual aspects of the integratedmanagement system under the command and control of informationtransmitted from the central computer system to the wireless wearableterminal.

As the workers complete their assigned tasks, a bi-directionalcommunication stream of information is exchanged over a wireless networkbetween wireless wearable terminals and the central computer system.Information received by each wireless wearable terminal from the centralcomputer system is translated into voice instructions or text commandsfor the corresponding worker. Typically, the worker wears a headsetcoupled with the wearable device that has a microphone for voice dataentry and an ear speaker for audio output feedback. Responses from theworker are input into the wireless wearable terminal by the headsetmicrophone and communicated from the wireless wearable terminal to thecentral computer system. Through the headset microphone, workers maypose questions, report the progress in accomplishing their assignedtasks, and report working conditions, such as inventory shortages. Usingsuch wireless wearable terminals, workers may perform assigned tasksvirtually hands-free without equipment to juggle or paperwork to carryaround. Because manual data entry is eliminated or, at the least,reduced, workers can perform their tasks faster, more accurately, andmore productively.

An illustrative example of a set of worker tasks suitable for a wirelesswearable terminal with voice capabilities may involve initiallywelcoming the worker to the computerized inventory management system anddefining a particular task or order, for example, filling a load for aparticular truck scheduled to depart from a warehouse. The worker maythen answer with a particular area (e.g., freezer) that they will beworking in for that order. The system then vocally directs the worker toa particular aisle and bin to pick a particular quantity of an item. Theworker then vocally confirms a location and the number of picked items.The system may then direct the worker to a loading dock or bay for aparticular truck to receive the order. As may be appreciated, thespecific communications exchanged between the wireless wearable terminaland the central computer system can be task-specific and highlyvariable.

To perform speech recognition, speech recognizer algorithms analyze thereceived speech input using acoustic modeling and determine the likelyword, or words, that were spoken (also known as the hypothesis). As partof the analysis and determination, the speech recognizer assignsconfidence factors that quantitatively indicate how closely each word ofthe hypothesis matches the acoustic models. If the confidence factor isabove the acceptance threshold, then the speech recognizer accepts thehypothesis as correctly recognized speech. If, however, the confidencefactor is below the acceptance threshold, then the speech recognizerrejects or ignores the speech input. This rejection may require the userto repeat the speech input. By rejecting the hypothesis and requiringrepetition of speech that was otherwise correctly recognized, this typeof speech recognizer may reduce productivity and efficiency and,thereby, may waste time and money.

Accordingly, there is a need, unmet by current speech recognizersystems, for a speech recognizer that reduces unnecessary repetition.There is further a need for a speech recognizer that can accept speechinput, under certain circumstances, even if the confidence factor isbelow the normal acceptance threshold, without sacrificing accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this Specification, illustrate embodiments of the invention and,together with the Detailed Description of the embodiments given below,serve to explain the principles of the invention.

FIG. 1 illustrates a schematic view of a speech recognition system.

FIG. 2 illustrates a flowchart of an exemplary algorithm for modifyingthe behavior of the acceptance threshold of a speech recognition systembased on knowledge about an expected response from a user.

FIG. 3 illustrates a flowchart of an exemplary algorithm for using theinvention in speech recognition systems where words are reportedindividually as they are recognized.

It should be understood that the appended drawings are not necessarilyto scale, presenting a somewhat simplified representation of variousfeatures illustrative of the basic principles of the invention. Thespecific design features of the sequence of operations as disclosedherein, including, for example, specific dimensions, orientations,locations, and shapes of various illustrated components, will bedetermined in part by the particular intended application and useenvironment. Certain features of the illustrated embodiments have beenenlarged or distorted relative to others to facilitate visualization andclear understanding. In particular, thin features may be thickened, forexample, for clarity or illustration.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

FIG. 1 illustrates a schematic view of a speech recognition system inaccordance with the present invention. One of ordinary skill in the artwill recognize that various of the functional blocks of the speechrecognition system, including the inventive features, can be implementedusing a variety of different technologies.

Referring to FIG. 1 in system 100, a speech signal, such as from asystem user, may be captured by a speech input device 102 in a varietyof conventional ways. Typically, a microphone or otherelectro-acoustical device senses speech input from a user and convertsit into an analog voltage signal 103 that then is forwarded to a signalprocessor 104. As is conventionally known, the signal processor 104includes the necessary analog-to-digital converters, filters, andequalization circuitry that converts the analog speech input 103 into adigitized stream of data 105 that can be separated into separate unitsfor analysis. Alternatively, this audio data from device 102 can beretrieved from a data storage device. As discussed herein, the system100 might be realized by suitable hardware and software. As such, theblocks shown in FIG. 1 are not meant to indicate separate circuits or tobe otherwise limiting, but rather show the functional features andcomponents of the system.

In particular, the signal processor 104 divides the digital stream ofdata that is created into a sequence of time-slices, or frames 105, eachof which is then processed by a feature generator 106, thereby producinga vector, matrix, or otherwise organized set of numbers 107 representingthe acoustic features of the frames. Further explanation of an exemplaryspeech signal processor system is provided in U.S. Pat. No. 4,882,757,entitled SPEECH RECOGNITION SYSTEM, the disclosure of which isincorporated herein, by reference, in its entirety. This patentdiscloses Linear Predictive Coding (LPC) coefficients to representspeech; however, other functionally equivalent methods are contemplatedwithin the scope of the present invention as well.

A speech recognition search algorithm function 108, realized by anappropriate circuit or software in the system 100 analyzes the featurevectors 107 in an attempt to determine what hypothesis to assign to thespeech input captured by input device 102. As is known in the art in onerecognition algorithm, the recognition search 108 relies onprobabilistic models from a library of suitable models 110 to recognizethe speech input 102. Some of the models in the library 110 may becustomized to a user via templates or other means, while some models maybe generic to all users.

When in operation, the search algorithm 108, in essence, compares thefeatures 107 generated in the generator 106 with referencerepresentations of speech, or speech models, in library 110 in order todetermine the word or words that best match the speech input from device102. Part of this recognition process is to assign a confidence factorfor the speech to indicate how closely the sequence of features from thesearch algorithm 106 matches the closest or best-matching models inlibrary 110. As such, a hypothesis, consisting of one or more vocabularyitems and associated confidence factors 111 is directed to an acceptancealgorithm 112. If the confidence factor is above a defined acceptancethreshold, then the recognized speech is accepted by the acceptancealgorithm 112. Acceptance algorithm 112 may also be realized by anappropriate circuit or software component of the system 100. If,however, the confidence factor is not above the acceptance threshold, asutilized by the acceptance algorithm, then the acceptance algorithm 112ignores or rejects the recognized speech or prompts the user to repeatthe speech. In this instance, the user may repeat the speech to inputdevice 102.

One common modeling technique utilized for speech recognition includesHidden Markov Models (HMM). In speech recognition, these models usesequences of states to describe vocabulary items, which may be words,phrases, or subword units. Each state represents one or more acousticevents and serves to assign a probability to each observed featurevector. Accordingly, a path through the HMM states produce aprobabilistic indication of a series of acoustic feature vectors. Themodel is searched such that different, competing hypotheses (or paths)are scored; a process known as acoustic matching or acoustic searching.A state S can be reached at a time T via a number of different paths.For each path reaching a particular state at a particular time, a pathprobability is calculated. Using the Viterbi algorithm, each paththrough the HMM can be assigned a probability. In particular, the bestpath can be assigned a probability. Furthermore, each vocabulary item orword in the best path can be assigned a probability. Each of theseprobabilities can be used as a confidence factor or combined with othermeasurements, estimates or numbers to derive a confidence factor. Thepath with the highest confidence factor, the hypothesis, can then befurther analyzed. The confidence factor of the hypothesis or theconfidence factors of each vocabulary item in the hypothesis can becompared with an acceptance threshold. As used herein, the term “word”is used to denote a vocabulary item, and thus may mean a word, a segmentor part of a word, or a compound word, such as “next slot” or “sayagain.” Therefore, the term “word” is not limited to just a single word.It should be understood that other speech recognition models are alsocontemplated within the scope of the present invention; for example,template matching dynamic time warping (DTW) and neural networks are twosuch exemplary, alternative modeling techniques.

While existing speech recognition systems adequately address the task ofdetermining the spoken input and recognizing the speech, there are stillsome drawbacks in such systems as noted above. For example, all of thehypotheses generated by the system, even the best-scoring hypothesis,may have confidence factors that fall below the acceptance threshold. Insuch a situation, the speech is rejected and may have to be repeated. Asnoted, this reduces productivity and efficiency. The present inventionaddresses such issues and improves upon the recognition efficiency byusing expected speech to modify the acceptance threshold.

More specifically, in certain environments utilizing speech recognitionsystems, the expected speech or expected response to be received from auser is known beforehand or can be determined. For example, when pickinga part from a bin, or placing an item in a warehouse location, there canbe a check-digit that verifies the location of the user or the operationbeing performed by the user. During the operation, the user is promptedto go to a location and speak the check-digit (or check-phrase)displayed at and associated with the location. The speech recognitionsystem attempts to recognize the user's spoken response and compare itwith this check-digit or check-phrase to confirm that the user is at thecorrect location before prompting the user to perform an operation, suchas picking a case of product. As used herein, “check-digit” refers tothe set of one or more words that are associated with a location, bin orslot for the purpose of verifying a user's location. A “check-digit”may, for example, be a three-digit number or could be non-digit words.In addition to this exemplary environment, there may be other scenariosin which a particular response or series of responses is expected fromthe user. Other such examples are described in US patent application2003/0154075 and include password verification, quantity verification,and repeat/acknowledge messages. The exemplary embodiments of thepresent invention described below involve situations wherein one or moreknown expected response(s) are utilized to improve speech recognitionsystems. In addition to these exemplary environments, an expectedresponse may be known in other situations when a recognizer is waitingfor a response from the user. As recognized by one of ordinary skill,the principles of the present invention can be advantageous within thesesituations as well.

In embodiments of the present invention, this knowledge about theresponse that is expected from the user is utilized to modify and refinea speech recognition system to increase efficiency and productivity. Inparticular, the speech recognizer, as described herein, uses theinformation about the expected response in addition to the conventionalmodels and probabilistic determination methods in order to accuratelydetermine what a user has said.

FIG. 2 illustrates a flowchart of an exemplary algorithm for modifyingthe behavior of an acoustic match/search process of a speech recognizerin accordance with aspects of the invention. In step 202, speech inputis received from the user by an appropriate device 102 or is pulled fromstorage. The speech input is then digitized in step 204, such as bysignal processor 104. As is known in the art, the digitized speech isseparated, in step 206, into time slices and converted into featurevectors, such as by feature generator 106. As explained earlier, thesefeature vectors correspond to information within an acoustic model thatallows a search and match to be performed to identify the most likely,or most probable, sequence of words spoken by the user. Step 208reflects the search/match routine to determine a hypothesis with thehighest associated confidence factor. Such a search/match routine may beperformed utilizing the speech recognition search algorithm 108 andmodel library 110 as shown in FIG. 1.

In accordance with the principles of the present invention, this mostprobable sequence, or the hypothesis with the highest confidence factor,is compared, in step 210, to an expected response that was knownbeforehand. Then, based upon such a comparison, the acceptance algorithmis modified. If the comparison shows that the most probable speechhypothesis matches an expected response, the hypothesis is morefavorably treated. Specifically, the acceptance threshold is modified bybeing downwardly adjusted or lowered in order to allow speech that mayotherwise be rejected to be accepted and output as recognized by system100. In one embodiment of the invention, as discussed herein, theassumption is that the recognizer uses higher or increased confidencescores to represent a higher confidence that the speech input 102actually matches the recognizer's hypothesis. In such a case theinvention would operate to decrease the acceptance threshold based uponknowledge of an expected response to allow recognition of the spokeninput. The present invention also contemplates however that theconfidence scale could be inverted. That is, the recognizer would uselower or low scores to represent higher confidence. In such a case, theconfidence scores must be below a defined acceptance threshold for therecognizer to accept the speech input. In this case, in accordance withaspects of the invention, the acceptance threshold would then beincreased (not decreased) to yield the same effect; namely, that thehypothesis or components of the hypothesis are more likely to beaccepted by the system.

Referring to FIG. 2, in step 210, the most likely sequence of words(highest confidence factor), as determined by the search 208, iscompared to the expected response. As noted above, the matchingalgorithm requires that a hypothesis is accepted only if its confidencefactor exceeds an acceptance threshold 214. As part of the invention, instep 212, the acceptance threshold is adjusted, such as by beinglowered, if the hypothesis matches the expected response. In such acase, the hypothesis 111 may initially have a confidence factor that istoo low to have the hypothesis accepted by acceptance algorithm 112, andthe operator would have had to repeat the response. However, inaccordance with the principles of the present invention, if theotherwise rejectable hypothesis compares favorably to the expectedresponse, the threshold is lowered. If the threshold is lowered enough,the most likely hypothesis is now recognized and accepted, in step 216,as recognized speech because its confidence factor exceeds the loweredacceptance threshold. There may still continue to be circumstances inwhich speech input received is not recognized even if prior knowledge ofan expected response is available and the speech compared well with theexpected response. For example, the threshold may not be lowered enoughfor the hypothesis to be accepted. In such an instance, the speech inputis rejected in step 218, and the user repeats the speech input.

An example of the invention is useful to illustrate its features. Forexample, a user may be directed to a particular bin or slot and asked tospeak the check-digits assigned to that bin or slot to verify hislocation in a warehouse. For the purpose of this example, we assume thatthe check-digit is “one one”. The acceptance threshold for the system isset to 1.5. Various scenarios may arise in this situation.

In the first scenario, the user speaks the correct check-digits and thesearch algorithm produces “one one” as the top hypothesis, with aconfidence factor of 2. In this case, the check-digits are acceptedbecause the confidence factor exceeds the acceptance threshold and theuser continues with the task.

In the second scenario, the user speaks the correct check-digits and thesearch algorithm again produces “one one” as the top hypothesis.However, in this later scenario, the hypothesis is only assigned aconfidence factor of 1. Without the invention, this hypothesis wouldnormally be rejected because the confidence factor is lower than theacceptance threshold of 1.5. The user would then be asked to repeat thespeech causing delay or inefficiency.

With the invention, the system adapts. Specifically, the system may knowthe expected check-digit response from the user based on knowledge ofthe bin or slot. The invention makes a comparison of the top hypothesisto the expected response for the user. If the hypothesis matches theexpected check-digit response for the user's location, the acceptancethreshold is lowered, such as to 0.5 for example. Now the confidencefactor (1) exceeds the acceptance threshold (0.5). The check-digitresponse of “one one” is then accepted and the user continues with thetask without having to repeat the check-digits. This change constitutessavings of time leading to higher efficiency and productivity.

In a third scenario, the search algorithm produces incorrectcheck-digits as its top hypothesis (either the user said the wrongcheck-digits or the speech was recognized incorrectly or the hypothesiswas produced due to background noise and not user speech), e.g. “onetwo”, with a confidence factor of 1. Since the hypothesis does not matchthe expected check-digits at the user's location (i.e., bin/slot), theacceptance threshold is not adjusted or lowered. Therefore, since theconfidence factor is below the acceptance threshold, the hypothesis isrejected. Therefore, the invention does not cause acceptance of thewrong response.

In a fourth scenario, the search algorithm produces incorrectcheck-digits as its top hypothesis (either the user said the wrongcheck-digits or the speech was recognized incorrectly or the hypothesiswas produced due to background noise and not user speech), e.g. “onetwo.” However, now the hypothesis has a confidence factor of 2. Sincethe confidence factor exceeds the rejection threshold (1.5), thehypothesis is accepted and the user is alerted that the check-digits areincorrect. As may be appreciated, there are numerous other situationswhere there is an expected response that would lead to adjusting theacceptance threshold. The example provides a single illustration of theinvention and is not meant to limit the situations where it may beuseful.

Thus, according to the method detailed in FIG. 2. Speech input is firstanalyzed and processed by the recognizer to determine the most likelymatch or hypothesis. Unlike previous methods that simply assign aprobability to this hypothesis, the present invention compares thehypothesis to a predetermined value (or values), such as the expectedresponse. If the expected response and the hypothesis are essentiallythe same, then the acceptance threshold for the hypothesis is adjustedsuch that speech recognition and acceptance is more likely.

There are a variety of ways to include knowledge about an expectedresult within a speech recognition application for the purposes of theinvention. For example, when developing the software, the developer mayinclude this information in tables or other data structures that arereferenced at different points in the execution of the application. Forexample, the program may know where in its execution it is to look for a“YES” or “NO” answer. Additionally, or alternatively, the informationabout the expected result can be calculated dynamically usingprogramming logic included in the speech recognition application. Forexample, it is well known that the accuracy of a credit card number canbe calculated based on a particular checksum algorithm. In such anexample, the speech recognition program would not need to have all thechecksums precalculated beforehand, but can implement the checksumalgorithm dynamically to calculate a value on-the-fly as needed. Inanother example, the program may know the location (bin/slot) that auser has been sent to and may then know the specific check-digits tolook for in the expected response. This on-the-fly information is stillavailable as “prior knowledge” with which to evaluate the speechreceived by a user, and thus the present invention may use pre-storedexpected responses or dynamically developed expected responses.Referring to FIG. 1, the expected response 114 may be used by theacceptance algorithm and may be retrieved from memory or calculateddynamically by system 100 in order to compare it to the hypothesis.

The amount by which the acceptance threshold is adjusted can bedetermined in various ways according to embodiments of the invention. Inone embodiment, the voice development tool or API used to implementsystem 100 can provide a means for the application developer to specifythe adjustment amount. For example, a fixed amount of thresholdadjustment 116 may be built into the system 100 and used by acceptancealgorithm 112 as shown in FIG. 1. Alternatively, the recognition systemor application can be configured to automatically calculate anappropriate adjustment amount to lessen the knowledge needed by theapplication developer and prevent inappropriate use of the features ofthe invention.

For example, in one embodiment, to guard against the expected responsebeing accidentally produced and accepted by the recognition system, theadjustment of the acceptance threshold may be dynamically controlled byan algorithm that takes into account the likelihood of the recognitionsystem accidentally producing the expected response. For example, in onesuch embodiment, the present invention contemplates at least twoindependent components of such a threshold adjustment algorithm: thenumber of vocabulary items at the point in the application where theexpected response is used (the breadth of the search), and the number ofvocabulary items in the expected response (the depth). For example, ifthere are only two possible responses (e.g., a “yes” or “no” response)with one being the expected response, then the adjustment to theacceptance threshold could be made very small because the recognizer islooking for a single word answer (depth=1) from only two possibilities(breadth=2).

Alternatively, in such a scenario, the system 100 could be configured toprovide no adjustment to the acceptance threshold, because with such alow depth and low breadth, there is a higher chance of the systemproducing the expected response by accident.

However, in another recognition scenario, if there are a hundredpossible responses (e.g., a two-digit check-digit), then the probabilityof producing the expected response by accident would be smaller and theadjustment to the acceptance threshold therefore may be made moresignificant. For example, two check-digits will have a hundred possibleresponses (10 possibilities×10 possibilities) making a breadth 100 and adepth of 2 for the two check-digits. This would allow a more significantthreshold adjustment to be used.

In another embodiment, the threshold adjustment may depend on how likelythe user is to say the expected response. If, in a particularapplication, the user says the expected response 99% of the time, thethreshold adjustment may be greater than in applications where theuser's response is not as predictable. For example, if the user goes tothe proper spot or location in a warehouse most of the time or is askedto speak a password, it may be desirable for the system to use a greateradjustment, because the response will usually be correct. However, ifthe system is less sure of the response to be received, less adjustmentto the threshold will be used to prevent improper acceptance of aresponse. The system 100 could thus dynamically vary the thresholdadjustment amount accordingly based on the likelihood of the user sayingthe expected response. The likelihood of the user saying the expectedresponse could be determined ahead of time or dynamically and/orautomatically estimated based on actual usage patterns.

For example, in the check-digits scenario described earlier, theprobability (Pe) that the invention causes the user to pick an item fromthe wrong slot by incorrectly accepting the hypothesis as the expectedresponse is given by,Pe=Pa*Pb*Pc*PdWhere:Pa is the probability that the user is at the wrong slot or shelf,Pb is the probability that the top hypothesis is not what the user said,Pc is the probability that the misrecognition matches the expectedresponse, andPd is the probability that the confidence factor is less than theacceptance threshold but higher than the adjusted acceptance thresholdfor incorrect hypotheses.

Pa, Pb, Pc, and Pd depend on many factors including (but not limited to)the following:

Pa: user, the site layout, etc.

Pb: the accuracy of the speech recognizer and the speech pattern of theuser

Pc: the depth and breadth of the grammar at the given instance

Pd: the distribution of the confidence scores, the acceptance threshold,and the acceptance threshold adjustment.

It is easy to see that to control Pe by means of the thresholdadjustment, the other probabilities need to be estimated. Pa, Pb, Pc,and Pd may be static quantities or may be adjusted dynamically as thesystem observes a given user's patterns.

For example, if the system notices that a user is more prone to go to anincorrect shelf, it would adjust its estimate of Pa higher. To maintainthe same Pe, the system would then change its adjustment to theacceptance threshold to make Pd lower. Therefore, the amount of thethreshold adjustment may be dynamically determined. This is just one ofmany possible ways and a person of ordinary skill in the art willappreciate that the other factors can also affect the thresholdadjustment according to the formula noted above.

In still another embodiment, the acceptance threshold adjustment amountcan also be determined by taking into account the “cost” of therecognizer making an error. A smaller adjustment would be used when thecost is greater, to prevent errors as described above where an incorrecthypothesis is mistakenly recognized as the expected response. Forexample, the invention could be used on the passcode for a rocket ormissile launcher or in a scenario to confirm a warehouse worker'slocation when picking health and beauty products. The “cost” of making amistake in the rocket example is much higher than in the warehouseexample. Thus, all other components/factors being equal, the adjustmentamount used for the rocket example should be chosen to be less than theadjustment for the warehouse example. In practice, this cost ofmistakenly recognizing the expected response must be balanced againstthe cost of requiring that the operator repeat the speech input 102.

In one embodiment of the invention, the cost of mistakenly recognizingthe hypothesis as the expected response can be expressed as:C _(e1,avg) =C _(e1) *PeWhere:C_(e1) is the cost for mistakenly recognizing the expected response, andC_(e1,avg) is the average cost for mistakenly recognizing the expectedresponse. This is compared with the average savings that the inventiongenerates by preventing the operator from having to repeat the speechinput when he's at the correct location and says the correctcheck-digits, S.S=C _(rep)*(1−Pa)*(1−Pb)*PfWhere Pa and Pb are as defined above.C_(rep) is the cost of the operator having to repeat himself, andPf is the probability that the confidence factor is between theunadjusted and adjusted acceptance thresholds for correct hypotheses.

In still another embodiment, the system 100, through the use of theinvention and a threshold adjustment, may be used to effectivelyoverride the acceptance algorithm. For example, in those scenarioswherein the cost of a wrong answer is insignificant in the context ofthe application, the reduction to the threshold can effectively beinfinite. That is, the recognizer would accept the hypothesis, (whichequals the expected response) regardless of how its confidence factorrelates to the threshold.

In accordance with another aspect of the invention, a related toolallows a voice application developer to specify the expected responsefor the system. (Herein, “voice application developer” or “developer” isused to refer to the user of the voice application development tool,i.e. the person who creates or programs the voice application. Thedeveloper is not to be confused with the user of the voice recognitionsystem 100.) Voice application development tools allow a voiceapplication developer to specify what vocabulary items and grammar areavailable to be recognized at a given point of a dialog. At least onesuch voice application development tool 101 exists separate and apartfrom the speech recognition system 100. The outputs of the tool 101 areused by the speech recognition system 100 to define the behavior of thespeech recognition system. For example, using an interface of the voiceapplication development tool, a developer specifies what information isto be delivered to a user of the speech recognition system, such as whatsentences or questions the system will prompt the user for. Thedeveloper also specifies what responses the speech recognizer shouldattempt to recognize. For example, using a voice application developmenttool 101, a developer can specify that the voice application prompts theuser “What is the check-digit?” and that the voice application shouldlisten for the vocabulary items “one”, “two”, “three”, “four”, “five”,or “six.” Referring to FIG. 1, system 100 may be designed such thatmodels for responses, which the user may say 110 are available to thesearch algorithm 108, or other components of the system at the properjuncture in the application dialog. In accordance with the invention,the tool also provides the developer the means to specify certainexpected responses 114 in the system. For the example above, the toolwould allow the developer to specify that the expected response is thecorrect check-digit (e.g. “one”). In another aspect, the voiceapplication development tool may be used to allow the operator tospecify one or more expected responses in the form of an expression orexpressions that the tool 101, speech recognition system 100 or voiceapplication knows how to evaluate to come up with expected response(s).The expressions may, for example, contain constants, mathematical orstring functions, table lookups, and application variables. As notedabove, with the invention, the voice application development toolinterface may also allow the voice application developer to specify thethreshold adjustment amount or any values used to calculate it for eachexpected response. The tool 101 has an appropriate processing functionfor processing information that is input by the developer through thetool interface to generate an output. The voice application developmenttool 101 stores at least one of the following in its output for use bythe speech recognition system 100: the specification of the expectedresponse (i.e. the expression(s) that are later evaluated to yield theexpected response), the adjustment amounts, and values that can be usedfor calculating the adjustment amounts. The system might use the latterto dynamically set the adjustment amount.

FIG. 3 illustrates another aspect of the present invention wherein theconfidence factors of words or other parts of a multi-word inpututterance are individually compared to an acceptance threshold. In thisembodiment, under certain circumstances, the decision on whether toaccept and report an individual word is delayed in order to takeadvantage of the expected response features of the invention.Specifically, the decision on whether to accept words whose confidencefactors are not initially above the acceptance threshold is delayeduntil it is known that either the hypothesis matches the completeexpected response or the hypothesis does not match the expectedresponse. In speech recognition systems where words are reportedindividually as soon as they are determined, this aspect of theinvention provides the benefit that words continue to be reported assoon as possible, minimizing system response time, except in thoseinstances when the expected response features of the invention takeeffect to help recognition performance. For example, it may be that athree or four word utterance has one word whose confidence factor doesnot immediately exceed the acceptance threshold, whereas other words inthe utterance have confidence factors that exceed the acceptancethreshold and are thus accepted. The invention provides acceptance ofthe hypothesis based upon the expected response even if one or morewords are initially rejected. While the example below utilizes athree-word utterance with only one word initially below the acceptancethreshold, the invention is not so limited.

Referring to FIG. 3, as an utterance is spoken, the speech recognitionsearch algorithm 108 generates individual words (or vocabulary items) ofa hypothesis, as set forth in step 302. For example, in response tobeing prompted to speak check-digits, a user might speak thecheck-digits 125 or “one,” “two,” “five”, and the search algorithm mightproduce a hypothesis consisting of the words “one”, “two”, and “five”,one word at a time. However, for any number of various reasons, thesystem 100 may not have a high confidence that the user spoke the word“two”. In accordance with the principles of the present invention, thesystem may know that the expected response for the check-digits is “one,two, five”, and may utilize the expected response feature of theinvention to ensure that the speech of the user is properly recognizedand accepted despite one or more words having a confidence factor orvalue that is below an acceptance threshold.

Referring again to FIG. 3, an utterance is spoken by a user and thesearch algorithm 108 produces a hypothesis consisting of individualvocabulary words. The system generates confidence factors for the wordsof the hypothesis. As the search engine produces each word, the systemcan queue or otherwise store each word for immediate or later processingas described below, as indicated by step 304. As the utterances begin,the system will generally be in a Reporting state, and thus flow willcontinue through step 306 to step 308. The first vocabulary word in theutterance “one two five” is the word “one.” For the portion of thehypothesis corresponding to the word “one,” the confidence factor iscompared to an acceptance threshold (step 308). In the present example,the word “one” has a sufficiently high confidence factor to exceed theacceptance threshold and, thus proceeds through step 310 as an acceptedword. As such, the system stays in the Reporting state (step 312) andthe word “one” is reported in step 314. The system then proceeds to waitfor the next word in the utterance as noted in step 316.

In the example, the utterance continues because three digits of thenumber 125 were spoken. With the recognition of the vocabulary word“two” the system queues or otherwise stores the vocabulary word “two”and its confidence factor. As the system is still in the Reportingstate, it thus proceeds through step 306 to step 308. However, in thisexample, the word “two”, with its low confidence factor as noted above,does not pass the acceptance criteria. Pursuant to step 310, the word“two” is not acceptable or is initially rejected. In such a case, inaccordance with the principles of the present invention, the systemprogresses on a path to step 318 so that the portion of the hypothesisor utterance string, which at this stage includes the words “one two,”is compared to the beginning portion of the expected response. Since theexpected response is “one two five”, the first parts or portion of thehypothesis compare favorably to the first parts or portion of theexpected response, such that the portion of the hypothesis generated sofar is considered to be part of the expected response, as noted in step318. Because the hypothesis is part of the expected response, theinvention progresses through step 320 to step 322. In step 322, thehypothesis generated so far is compared to the expected response in itsentirety. Since the hypothesis “one two” does not match the completeexpected response “one two five”, the flow proceeds to step 330, wherethe system is switched to a “not reporting” state. Then the system waitsfor the next vocabulary word of the hypothesis in step 316.

The system waits in step 316 until the next word is received. In theexample, the next word is “five.” The vocabulary word encounters thissystem now in the Not Reporting state (step 306). Therefore, the systemprogresses to step 318 where the hypothesis string, that now includesthe words “one, two, five”, is compared to the beginning portion of theexpected response, which is “one two five”. Since the utterance stringis part of the expected response pursuant to step 320, the nextdetermination to be made is whether the complete expected response hasbeen received. In this example, the last word “five” completes theexpected response and, thus, pursuant to step 322, the system proceed tostep 324 wherein the acceptance threshold is lowered or otherwiseappropriately adjusted, and the confidence factor of each queued word iscompared to the adjusted threshold (step 326). In one example, thelowering of the acceptance threshold may have been sufficient so thatthe spoken “two” was now accepted. Therefore, all words “one, two, five”are accepted. Alternatively, if the threshold was not lowered enough,then the “two” would not be accepted. The threshold is reset to itsoriginal value in step 328, and pursuant to step 312, the systemswitches to a Reporting state. In step 314, words that have been queuedand are now accepted are reported as recognized words. In the examplejust given, because the utterance compared favorably to the expectedresponse and the confidence factor of each word of the hypothesis metthe adjusted acceptance threshold, the entire utterance (e.g., the threewords “one two five”) is reported as being accepted.

In an alternative example, one or more of the words might not be part ofthe expected response. For example, the middle word might have beenheard and recognized as “three” rather than “two.” In this case, theutterance detected or hypothesis is “one three five.” If “one” isinitially accepted and “three” is not accepted because it has a lowconfidence factor, then flow would proceed through steps 308 and 310 tostep 318. Then, pursuant to a comparison to the expected response (step318), it would be determined that the partial hypothesis “one three” wasnot part of the expected response (step 320). Then, the system would notadjust the threshold, switch to a “not reporting” state, nor delay thedecision on this word. As such, if a word is initially rejected and isnot part of the expected response, the word would not be accepted. Ifthe last word spoken was actually “five” but was initially rejected and“one two” has been accepted, the word “five” may be stored according tothe principles of the invention (step 304) and thereafter re-evaluatedagainst a lower threshold value (step 326). In this example, the “five”is only stored long enough to test it with the adjusted threshold, sinceit matches the last word of the expected response. Although one featureof the invention is that the reporting of accepted words is sometimesdelayed, this example illustrates that the invention does not alwaysdelay the reporting of accepted words.

As such, the present invention is able to analyze individual words of amulti-word utterance and instead of initially rejecting all or part ofthe utterance because one or more words are not initially accepted, itis able to queue or store the unaccepted words and then make a latercomparison or analysis based upon how the utterance matches against theexpected response string. In that way, the invention is able to acceptthe properly spoken expected response even though one or more words haveconfidence factors that do not compare favorably against the initialacceptance threshold, but do compare favorably against the adjustedthreshold.

Thus, while the present invention has been illustrated by a descriptionof various embodiments and while these embodiments have been describedin considerable detail, it is not the intention of the applicants torestrict or in any way limit the scope of the appended claims to suchdetail. Additional advantages and modifications will readily appear tothose skilled in the art. Thus, the invention in its broader aspects istherefore not limited to the specific details, representative apparatusand method, and illustrative example shown and described. Accordingly,departures may be made from such details without departing from thespirit or scope of applicants' general inventive concept. For example,the exemplary speech recognition system described herein has focused onwearable wireless terminals. However, the principles of the presentinvention are applicable to other speech recognition environments aswell.

What is claimed is:
 1. A method for recognizing speech, the methodcomprising the steps of: analyzing speech input with an apparatus usingat least one hardware-implemented processor to generate a hypothesisincluding multiple elements; generating respective confidence factorsfor the multiple elements of the speech input; comparing a confidencefactor for at least one of the elements to a threshold for accepting theelement as properly recognized; and if the element is not accepted inthe comparison, comparing at least a portion of the hypothesiscontaining the unaccepted element to at least a portion of an expectedresponse; if the comparison of the portion of the hypothesis containingthe unaccepted element with the portion of the expected response isfavorable, storing the unaccepted element for subsequent analysis andproceeding to further analyze another element of the hypothesis.
 2. Themethod of claim 1 wherein the elements include one or more words thatmake up the speech input.
 3. The method of claim 1 wherein each elementis an individual word.
 4. The method of claim 1 further comprising,after storing at least one unaccepted element for subsequent analysis,adjusting the acceptance threshold and comparing the confidence factorfor at least one stored element to the adjusted acceptance threshold. 5.The method of claim 1 further comprising, if the comparison of theportion of the hypothesis containing the unaccepted element with theportion of the expected response is not favorable, rejecting at least aportion of the hypothesis.
 6. The method of claim 5 further comprisingproceeding to analyze elements of the hypothesis until the fullhypothesis is analyzed or a portion of the hypothesis is rejected. 7.The method of claim 1 further comprising the step of either increasingor decreasing the acceptance threshold based on the favorable comparisonof the hypothesis portion to the expected response.
 8. The method ofclaim 1, further comprising comparing the confidence factors of elementsin the hypothesis to an acceptance threshold in a sequential fashion. 9.The method of claim 4 wherein an adjustment amount for the acceptancethreshold is a fixed amount.
 10. The method of claim 4 wherein anadjustment amount for the acceptance threshold is a variable amount. 11.The method of claim 1 further comprising generating the hypothesis in anelement-by-element fashion, and the subsequent analysis for a storedelement being performed after generating the next element in thehypothesis.
 12. A system for recognizing speech, comprising: ahardware-implemented processor that has a speech recognizer configuredto analyze speech input and generate a hypothesis of multiple elementsand respective confidence factors for the multiple elements; anacceptance algorithm that compares the confidence factor of at least oneelement to an acceptance threshold for accepting or not accepting theelement; an adjustment component that compares at least a portion of thehypothesis containing an element that is not accepted to at least aportion of an expected response; and if the comparison of the portion ofthe hypothesis containing the unaccepted element with the portion of theexpected response is favorable, the system operable for storing theunaccepted element for subsequent analysis and proceeding to furtheranalyze another element of the hypothesis.
 13. The system of claim 12wherein the elements include one or more words that make up the speechinput.
 14. The system of claim 12 wherein each element is an individualword.
 15. The system of claim 12 wherein the adjustment component, afterat least one unaccepted element is stored for subsequent analysis,adjusts the acceptance threshold and compares the confidence factor forat least one stored element to the adjusted acceptance threshold. 16.The system of claim 12 wherein the system rejects at least a portion ofthe hypothesis if the comparison of the portion of the hypothesiscontaining the unaccepted element to the portion of the expectedresponse is not favorable.
 17. The system of claim 12 wherein the systemgenerates the hypothesis in an element-by-element fashion, and thesubsequent analysis for a stored element is performed after the speechrecognizer generates the next element in the hypothesis.
 18. A methodfor recognizing speech, the method comprising the steps of: analyzingspeech input with an apparatus using at least one hardware-implementedprocessor to generate a hypothesis of multiple elements and respectiveconfidence factors for the multiple elements; comparing each saidconfidence factor to a threshold for accepting the element; andcomparing at least a portion of the hypothesis to at least a portion ofan expected response; if the comparison is favorable, adjusting athreshold for one or more of the elements in order to affect theacceptance of the one or more elements.